FE-1604: Allow Voice interruption by speaking - #9550
Conversation
Signed-off-by: Lu Nelson <ln@hash.ai>
The built-app test still required the retired Brunch identity sentence, so its faux provider rejected the request before emitting any response and surfaced an empty-reasoning assertion instead.
The production-path test already proves skill activation, resource reads, server and client tools, and transcript behavior. Exact system-prompt sentences only coupled it to editorial changes and obscured failures behind empty downstream output.
Signed-off-by: Lu Nelson <ln@hash.ai>
Signed-off-by: Lu Nelson <ln@hash.ai>
…rchitecture pinned
Add a software-behavior/Gherkin paper plugin, extract the cross-plugin transformation and evidence invariant, and capture the prospective software-correctness/Dafny boundary for mission planning. Amp-Thread-ID: https://ampcode.com/threads/T-01a05d1a-c858-722b-b53a-d0c448d14155 Co-authored-by: Amp <amp@ampcode.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
ae46593 to
435d7dd
Compare
Amp-Thread-ID: https://ampcode.com/threads/T-01a0772c-6fce-752a-9778-480fa5209c5c Co-authored-by: Amp <amp@ampcode.com>
PR SummaryHigh Risk Overview Agent composition moves SDCPN modelling skills, Petrinaut tools, and prompts into A Pi persona evaluation harness adds
Reviewed by Cursor Bugbot for commit f69ac17. Bugbot is set up for automated code reviews on this repo. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| this.#activeSubmission || | ||
| !this.#chat.canAcceptInterviewAnswer || | ||
| this.#chat.status !== "ready" | ||
| ) { |
There was a problem hiding this comment.
Speech request still drops interrupting answers
Medium Severity
canonical-speech-requested still marks every accepted input item as playback-overlapping, even when interruption is enabled. The session keeps that item accepted and emits the completed transcript, so the words can appear, but the bridge then rejects them as unavailable with no notice. The controller also still clears partialText on speech request and output start, so an in-flight utterance can vanish if more canonical speech is queued after the user already started talking.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 0666d5b. Configure here.
There was a problem hiding this comment.
Fixed in 69cde87. Interruption-originated input items now remain accepted across later canonical speech requests, and the controller preserves their in-flight partial text through speech request and output-start events. Added bridge and controller regressions.
| this.#transcriptItemId = null; | ||
| this.#transcriptKey = null; | ||
| this.#update({ inputNotice: "none", partialText: "" }); | ||
| return; |
There was a problem hiding this comment.
False rejection clears retained-answer notice
Low Severity
A later prompt-regurgitation or self-echo rejection always clears inputNotice and partialText. If an interrupting answer is already retained, the dock drops Answer captured. Waiting for Brunch. even though #pendingInterruption remains and will still submit. The user loses confirmation that the earlier answer is waiting.
Reviewed by Cursor Bugbot for commit 0666d5b. Configure here.
There was a problem hiding this comment.
Fixed in 69cde87. Rejected prompt-regurgitation and self-echo transcripts no longer clear a previously retained answer or its answer-pending notice, including across the later transcript event lifecycle.
b53b100 to
be56a18
Compare
Co-authored-by: Cursor <cursoragent@cursor.com>
github-actions
Bot
added
area/deps
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 69cde87. Configure here.
| this.#snapshot.inputNotice === "answer-pending" | ||
| ? {} | ||
| : { inputNotice: "none", partialText: "" }, | ||
| ); |
There was a problem hiding this comment.
Later failures clobber retained answers
Medium Severity
A later false interruption that finishes empty or failed still replaces an answer-pending retained answer with not-heard and clears partialText. Only prompt-regurgitation and self-echo keep the waiting state, so a common failed VAD can hide a queued answer that will still submit.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 69cde87. Configure here.
There was a problem hiding this comment.
Fixed in f69ac17. When an answer is already retained, later rejected false transcripts preserve its text and answer-pending notice; the transcription-failed session path now preserves it as well. Added red/green regressions for both empty and failed transcript event orderings.
Co-authored-by: Cursor <cursoragent@cursor.com>
2 participants
Important
Current status: #9537 and #9564 are merged. Wait for the omitted settlement port in #9588 to be accepted, then recut this draft from the latest
mainas an independent sibling PR. Do not merge it meanwhile. FE-1604 medium VAD and interruption behavior remain this PR’s separate follow-up delta.🌟 What is the purpose of this PR?
This PR lets a person interrupt Voice assistant playback by speaking without losing the interrupting utterance. It keeps the existing Your turn handoff as an optional half-duplex fallback and locally rejects likely false interruption transcripts before they become answers.
The change is stacked directly on #9531, which owns the shared Brunch Voice session, completed-transcript authority, canonical playback, and acknowledged manual handoff.
🔗 Related links
🚫 Blocked by
voice-interview-control.test.tsx🔍 What does this change?
speech_started → response.cancel → output_audio_buffer.clear. Interruption never clears the input buffer, and transcript validation does not gate cancellation.Completed-transcript hardening
A false VAD event can cause
gpt-4o-transcribeto repeat its vocabulary prompt or transcribe assistant playback. The Realtime bridge now runs a deterministic classifier only for completed interruption-originated transcripts, before retaining or submitting them:The transcription model and prompt remain unchanged. The prompt is shared between provider configuration and local validation to avoid drift. No extra model call, acoustic processing, debounce, or cancellation delay is introduced.
Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
📜 Does this require a change to the docs?
libs/@hashintel/petrinaut/docs/ai-assistant.md,apps/petrinaut-website/README.md, andlibs/@hashintel/brunch-agent/docs/adr/0009-openai-voice-ui-turn-shell.mddocument the Voice behavior and policy.🕸️ Does this require a change to the Turbo Graph?
voice-interview-control.test.tsx. These were reproduced before editing at the prior PR head, and the same six tests fail after hardening. This change does not fix or suppress them.localStorageunavailable. These are separate from the Voice changes and did not appear in the prior PR CI run.🛡 What tests cover this?
The prompt-leak, canonical-echo, queued-speech, retained-answer, and failed-transcript regressions were written first and observed failing before implementation. Coverage includes immediate cancellation, short novel answers, unchanged ordinary capture, duplicate completions, delayed/pending admission, active-playback-only snapshots, follow-on canonical speech, retained-answer display, and lifecycle cleanup.
Current verification at
f69ac17034:Focused Voice suites: 220 passed:
yarn workspace @apps/petrinaut-website test:unit src/main/app/voice-interview src/server/voice src/voice-diagnostics.test.ts --exclude '**/voice-interview-control.test.tsx'The known-failing control suite is excluded only from this focused command; it is included in the full run below.
yarn workspace @apps/petrinaut-website test:unit: 355 passed, 8 failed. Six failures are the unchanged maximum-update-depth cases above; two are the local-runtimelocalStoragefailures above.yarn workspace @apps/petrinaut-website build: passed.yarn workspace @apps/petrinaut-website lint:tsc: passed.yarn workspace @apps/petrinaut-website lint:eslint: passed with zero errors and one existingset-state-in-effectwarning in the unchanged control component.yarn workspace @local/petrinaut-arch-docs lint:arch-docs: passed.yarn lint:formatandgit diff --check: passed.Earlier PR verification recorded 618/618 Petrinaut library unit tests and passing library TypeScript checks. Those library checks were not rerun for this website-only runtime hardening; the library change in this increment is documentation only.
❓ How to test this?
🎥 UX capture
This recording shows the default-on Interruption by speaking option, the Your turn fallback appearing when it is disabled, and the fallback disappearing when it is re-enabled.
fe-1604-voice-interruption-by-speaking.mp4